POCO serialization#284
Conversation
|
Sorry, I created a merge conflict by checking in the Parquet variant projects :(. |
There was a problem hiding this comment.
Pull request overview
This PR adds a new POCO serialization subsystem for Apache Arrow in .NET, resolving issue #186. It provides two serialization paths: a source-generator-based AOT-safe approach using [ArrowSerializable] attributes and a reflection-based RecordBatchBuilder for anonymous types/prototyping.
Changes:
- New
Apache.Arrow.Serializationruntime library with attributes, interfaces (IArrowSerializer<T>,IArrowConverter<T>), helper classes, reflection-basedRecordBatchBuilder, and extension methods for Arrow IPC serialization - New
Apache.Arrow.Serialization.GeneratorRoslyn incremental source generator that emits schema derivation, serialization, deserialization code including polymorphic type support and JSON schema emission - Comprehensive test suite covering primitives, collections, nested types, enums, polymorphism, custom converters, callbacks, datetime types, diagnostics, and the reflection-based builder
Reviewed changes
Copilot reviewed 19 out of 20 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/Apache.Arrow.Serialization/Attributes.cs |
Defines all serialization attributes (ArrowSerializable, ArrowField, ArrowType, ArrowIgnore, ArrowMetadata, ArrowPolymorphic, ArrowDerivedType) and callback interface |
src/Apache.Arrow.Serialization/IArrowSerializer.cs |
IArrowSerializer<T> interface with static abstract members and IArrowConverter<T> for custom converters |
src/Apache.Arrow.Serialization/ArrowArrayHelper.cs |
Utility methods for building null arrays, Guid/TimeOnly/TimeSpan/Decimal arrays, and DateTime normalization |
src/Apache.Arrow.Serialization/ArrowSerializerExtensions.cs |
Extension methods for IPC byte/stream serialization and collection convenience methods |
src/Apache.Arrow.Serialization/RecordBatchBuilder.cs |
Reflection-based serializer for anonymous types and non-attributed objects |
src/Apache.Arrow.Serialization/README.md |
Comprehensive documentation covering all features |
src/Apache.Arrow.Serialization/Apache.Arrow.Serialization.csproj |
Runtime library project (net8.0) |
src/Apache.Arrow.Serialization.Generator/ArrowSerializerGenerator.cs |
Main incremental generator: type analysis, diagnostics, and orchestration |
src/Apache.Arrow.Serialization.Generator/CodeEmitter.cs |
Emits serialization/deserialization code for [ArrowSerializable] types |
src/Apache.Arrow.Serialization.Generator/PolymorphicCodeEmitter.cs |
Emits code for [ArrowPolymorphic] type hierarchies |
src/Apache.Arrow.Serialization.Generator/JsonSchemaEmitter.cs |
Emits optional JSON schema descriptors |
src/Apache.Arrow.Serialization.Generator/Models.cs |
Internal model classes for the generator pipeline |
src/Apache.Arrow.Serialization.Generator/Apache.Arrow.Serialization.Generator.csproj |
Generator project (netstandard2.0) |
test/Apache.Arrow.Serialization.Tests/SerializationTests.cs |
Tests for source-generated serialization round-trips |
test/Apache.Arrow.Serialization.Tests/RecordBatchBuilderTests.cs |
Tests for reflection-based builder |
test/Apache.Arrow.Serialization.Tests/DiagnosticTests.cs |
Tests for generator diagnostic reporting |
test/Apache.Arrow.Serialization.Tests/TestTypes.cs |
Shared test type definitions |
test/Apache.Arrow.Serialization.Tests/Apache.Arrow.Serialization.Tests.csproj |
Test project |
Directory.Packages.props |
Adds CodeAnalysis package versions |
Apache.Arrow.sln |
Adds new projects to solution |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| <PackageVersion Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" /> | ||
| <PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="4.11.0" /> | ||
| <PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="8.0.0" /> |
| public void Append(object? value) | ||
| { | ||
| if (value is null) _b.AppendNull(); | ||
| else _b.Append(new DateTimeOffset((DateTime)value, TimeSpan.Zero)); |
| // For null slots, we need a stand-in value (first non-null item) | ||
| object? standIn = _items.FirstOrDefault(v => v is not null); | ||
| foreach (var item in _items) | ||
| typedList.Add(item ?? standIn!); |
| Line($"if ({access} is {{ }} v_{index}) {{ bld_{index}_idx.Append((short)bld_{index}_dict.Count); bld_{index}_dict.Add(v_{index}.ToString()); }} else bld_{index}_idx.AppendNull();"); | ||
| else | ||
| { | ||
| Line($"bld_{index}_idx.Append((short)bld_{index}_dict.Count);"); | ||
| Line($"bld_{index}_dict.Add({access}.ToString());"); | ||
| } |
| // fall back to AppendNull for now — these are less common in polymorphic scenarios | ||
| Line($"bld_{index}.AppendNull(); // TODO: complex type {prop.Type.Kind}"); |
| break; | ||
| } | ||
| default: | ||
| Line($"object? prop_{propIndex} = null; // TODO: unsupported type {prop.Type.Kind}"); |
| // Remove trailing newline, add comma | ||
| sb.Length -= sb.ToString().EndsWith("\r\n") ? 2 : 1; |
| <TargetFramework>net8.0</TargetFramework> | ||
| <Nullable>enable</Nullable> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Description>Source-generated Apache Arrow serialization for .NET. Provides [ArrowSerializable] attribute and IArrowSerializer<T> interface for compile-time Arrow schema derivation, serialization, and deserialization.</Description> |
There was a problem hiding this comment.
I think we don't need to worry about net6.0 as it's out of support and we'll probably remove it as a build target after the next release. The inability to use with net472 or netstandard2.0 is a greater loss and it might be worth a quick test to see how hard it would be to add support for those.
There was a problem hiding this comment.
Thanks for the review! I'll take a look at your comments over the next few days and follow up.
There was a problem hiding this comment.
Thanks, this is great and has long been missing.
Can you please fix the merge conflict and the white space that our linter doesn't like? I think the Documentation task failure can be addressed by editing ci/scripts/docs.sh and doing something like
pushd "${source_dir}/src/Apache.Arrow.Serialization"
dotnet build -c Release
popd
before trying to build the documentation.
We should probably also add validation for the release by adding something to dev/release/verify_rc.sh like
reference_package "Apache.Arrow.Serialization" "Apache.Arrow.Serialization.Tests"
but I'm not sure the reference_package will handle the PrivateAssets="all" in the project file. If it doesn't, we could consider figuring that out later after the bulk of the change is checked in.
|
|
||
| namespace Apache.Arrow.Serialization.Tests; | ||
|
|
||
| public class DiagnosticTests |
| Global | ||
| GlobalSection(SolutionConfigurationPlatforms) = preSolution | ||
| Debug|Any CPU = Debug|Any CPU | ||
| Debug|x64 = Debug|x64 |
There was a problem hiding this comment.
It would be nice to avoid adding all these targets. Is there something bitness-specific in these changes?
| { | ||
| var list = items as IReadOnlyList<T> ?? items.ToList(); | ||
| if (list.Count == 0) | ||
| throw new ArgumentException("Cannot infer schema from empty collection.", nameof(items)); |
There was a problem hiding this comment.
Is this really true though? We use the type to infer the schema, not the data. It would be annoying for someone to have to special case an empty list if they want to serialize it.
| builders.Add(CreateColumnBuilder(propType, arrowType)); | ||
| } | ||
|
|
||
| var schema = new Schema.Builder(); |
There was a problem hiding this comment.
Consider moving schema above the foreach and adding the fields directly into the schema builder instead of a temporary list.
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFramework>net8.0</TargetFramework> |
There was a problem hiding this comment.
What would it take to make this work for .NET 4.7.2? Is that even plausible?
There was a problem hiding this comment.
the library now targets netstandard2.0;net462;net8.0 (matching Apache.Arrow), with tests passing on both net8.0 (197) and net472 (182 — the gap is only DateOnly/TimeOnly/Half, which don't exist there). To make that possible, IArrowSerializer changed from static abstract members (which require the .NET 8+ runtime) to an instance interface resolved through a generated, module-initializer-populated registry — the MessagePack IMessagePackFormatter pattern — so the public extension API is identical on every TFM, still reflection-free and AOT-safe.
| | `float` | `Float32` | | | ||
| | `double` | `Float64` | | | ||
| | `Half` | `Float16` | | | ||
| | `decimal` | `Decimal128(38, 18)` | Configurable via `[ArrowType("decimal128(28, 10)")]` | |
There was a problem hiding this comment.
It might be worth pointing out in the documentation that a CLR decimal is not a perfect match for an Arrow decimal.
|
|
||
| namespace Apache.Arrow.Serialization.Generator | ||
| { | ||
| internal enum TypeKind2 |
There was a problem hiding this comment.
Consider a more descriptive name. What about ArrowTypeKind?
There was a problem hiding this comment.
All open issues should be addressed.
Variant integration — separate PR or fold in here?
Since Variant landed in Apache.Arrow, I've prototyped a full serializer support for it on a local branch stacked on this one (~9 commits, 231 tests on net8.0 / 216 on net462. Before pushing anything: would you prefer this as a follow-up PR once this one merges, or folded into this PR? My instinct is follow-up — this PR is large enough — but happy either way.
A quick tour, in three layers:
1. VariantValue properties map to the Variant extension type — a column that holds any shape per row:
using Apache.Arrow.Scalars.Variant;
[ArrowSerializable]
public partial record Event
{
public string Name { get; init; } = "";
public VariantValue Payload { get; init; }
}
var evt = new Event
{
Name = "click",
Payload = VariantValue.FromObject(new Dictionary<string, VariantValue>
{
["x"] = VariantValue.FromInt32(10),
["tags"] = VariantValue.FromArray(VariantValue.FromString("a"), VariantValue.FromString("b")),
}),
};
var restored = Event.FromRecordBatch(Event.ToRecordBatch(evt));
int x = restored.Payload.AsObject()["x"].AsInt32(); // IPC round-trips too2. [ArrowType("variant")] stores any POCO/collection property as a self-describing Variant column instead of Struct/List — fully source-generated, recursive, no reflection:
[ArrowSerializable]
public partial record Order
{
public Address Shipping { get; init; } = new(); // Struct column (default)
[ArrowType("variant")]
public Address? Billing { get; init; } // Variant column
[ArrowType("variant")]
public Dictionary<string, double> Prices { get; init; } = new();
}On read, missing fields default and unknown fields are ignored — old readers survive new writers and vice versa.
3. An alternative wire format for [ArrowPolymorphic] — instead of the flat discriminator + union-of-all-fields schema, one Variant column with self-describing rows:
[ArrowPolymorphic(Format = ArrowPolymorphicFormat.Variant)]
[ArrowDerivedType(typeof(Order), "order")]
[ArrowDerivedType(typeof(Ping), "ping")]
public abstract partial record Message;
// wire:
// {"$type":"order","Id":"A1","Shipping":{"City":"Berlin"},"Tags":["a","b"]}
// {"$type":"ping","Seq":7}The schema never widens as derived types are added, and the flat union's property-type restrictions disappear (nested records, collections, and dictionaries all work). Flat union stays the default — it's the right choice for columnar analytics; the README has a trade-off table. There are also opt-in read policies (MissingFieldHandling / TypeMismatchHandling) for consuming foreign-written batches.
One piece you may want in this PR regardless: while testing IPC round-trips I found that extension columns don't rehydrate after ArrowStreamReader — ExtensionTypeRegistry.Default starts empty, so the generated (GuidArray) cast for the existing uuid mapping fails on any deserialized stream. The variant branch fixes this by registering the uuid (and variant) definitions from a module initializer (never overwriting existing registrations), plus a Guid IPC regression test. Happy to cherry-pick just that fix into this PR now if you'd like.
Adds Apache.Arrow.Serialization with a Roslyn incremental source generator that emits compile-time Arrow schema derivation, serialization, and deserialization for types marked with [ArrowSerializable]. - Runtime library (Apache.Arrow.Serialization): attributes, helpers, IPC extension methods, reflection-based RecordBatchBuilder - Source generator (Apache.Arrow.Serialization.Generator): code emission for 31+ type mappings, polymorphism, custom converters, callbacks - Test suite: 197 tests covering all supported types and features - Integrated into solution, central package management, Apache 2.0 headers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align with upstream CI which uses .NET 8.0 SDK. All 197 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove src/test solution folders from sln so serialization projects appear at root level like all other projects. Change serialization library target from net8.0;net10.0 to net8.0 for CI compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename TypeKind2 enum to ArrowTypeKind (review suggestion) - RecordBatchBuilder: allow empty collections (schema is inferred from the type, not the data) and add fields directly to Schema.Builder - RecordBatchBuilder: convert non-UTC DateTime kinds via ToUtcDateTimeOffset instead of misinterpreting local time as UTC - RecordBatchBuilder: handle all-null struct columns by creating a default stand-in instance instead of NullReferenceException - PolymorphicCodeEmitter: deduplicate enum dictionary entries so repeated values share one index - PolymorphicCodeEmitter: throw NotSupportedException for unsupported complex property types instead of silently serializing null - Pack the source generator into the Apache.Arrow.Serialization nupkg (analyzers/dotnet/cs) and add the package README - docs.sh: build the serialization library before docfx metadata - verify_rc.sh: verify the Apache.Arrow.Serialization package - Directory.Packages.props: restore alphabetical package ordering - README: document CLR decimal vs Arrow Decimal128 mismatch - Run dotnet format over the serialization projects Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
c6c05ba to
922cb4a
Compare
Multi-target Apache.Arrow.Serialization as netstandard2.0;net462;net8.0 (matching Apache.Arrow) so the serializer works on .NET Framework 4.6.2+ including 4.7.2. The static-abstract IArrowSerializer<T> interface required .NET 7+ runtime support and could not be polyfilled. It is redefined as an instance interface (the resolver pattern used by MessagePack-CSharp and System.Text.Json): the generator now emits, per type, a serializer class delegating to the generated statics and registers it with the new ArrowSerializerRegistry via [ModuleInitializer]. Resolution is one static-generic field read; no reflection, still AOT-safe. The generated statics on each type are unchanged and remain directly usable. - Add IArrowSerializable marker, implemented by generated partials; the generic extension methods keep their exact signatures on all TFMs with a `where T : IArrowSerializable` constraint and registry dispatch - Generator emits an internal ModuleInitializerAttribute polyfill when the consumer compilation lacks an accessible one (referenced assemblies such as Microsoft.CodeAnalysis carry inaccessible internal copies, so the probe checks accessibility, not mere existence) - Replace emitted Enum.Parse<T> (missing on .NET Framework) with an ArrowArrayHelper.ParseEnum<T> helper - Guard DateOnly/TimeOnly/Half code paths with NET6/NET5 conditionals, following the same pattern as Apache.Arrow itself; polyfill RequiresUnreferencedCode on downlevel TFMs - Tests multi-target net8.0;net472 on Windows: 197 pass on net8.0, 182 on net472 (DateOnly/TimeOnly/Half tests are #if-gated) - README: document target frameworks, .NET Framework prerequisites, and the registry-based dispatch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RC Verify job iterates every directory under src/ and fetches a matching nupkg; Apache.Arrow.Serialization.Generator is not packable (it ships inside the Apache.Arrow.Serialization package as an analyzer), so skip it in the artifact fetch loop. Add Apache.Arrow.Serialization.Tests to Apache.Arrow.Tests.slnf so the CI test job and RC verification actually run the serialization tests. During binary verification the generator project reference is dropped in favor of the packaged analyzer, and the generator unit tests (DiagnosticTests) are removed there because they need the generator assembly as a compile-time reference, which a packaged analyzer cannot provide; they still run against the source tree. Include the Apache.Arrow.Serialization README in the docfx site as a conceptual "Serialization" page (docs/_site/serialization/) with a toc entry and a link from the landing page. Verified locally with "docfx metadata/build --warningsAsErrors": 0 warnings, 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docfx metadata loads projects with Configuration=Release, so the source generator referenced as an analyzer only resolves from the Release output path. The plain (Debug) build left the analyzer unresolved, producing a FailedToResolveAnalyzer warning that --warningsAsErrors turned into the Documentation job failure (docfx exit code 255). Reproduced in a fresh clone: cold Debug build fails metadata with the warning; cold Release build passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
I think that makes sense.
Can you clarify this with a specific code sequence? There are constructors which take an |
Resolves #186.
I needed Arrow POCO serialization for an internal cross-language interop project (C# ↔ vgi-rpc-python). I took inspiration from
System.Text.Json's source generator andMessagePack-CSharp's attribute model, iterated on it with Claude as a coding assistant, and arrived at this implementation.Figured it might be useful upstream — please take a look and let me know what you think.
See README.md for full documentation and examples.